Corneal topography mapping with dense illumination

ABSTRACT

Techniques are described for generating and using an illumination pattern for corneal topography. The illumination pattern is projected onto an eye of a user wearing a head-mounted assembly. The illumination pattern is based on a reference pattern and corresponds to selective illumination of dots arranged along a two-dimensional grid. An image sensor captures a reflected image produced by reflection of the illumination pattern off the eye. A reflected pattern is identified based on glints in the reflected image and mapped to the reference pattern to generate an aligned reflected pattern. An eye model including a topography of a cornea is calculated by comparing the aligned reflected pattern to the reference pattern to determine a deviation in a shape of the cornea based on a difference between the aligned reflected pattern and the reference pattern. The eye model can be applied in various ways, including for eye tracking or biometric authentication.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of and priority to U.S. ProvisionalApplication No. 62/796,031, filed Jan. 23, 2019, entitled “CornealTopography Mapping With Dense Illumination.” The content of U.S.Provisional Application No. 62/796,031 is incorporated herein byreference in its entirety.

BACKGROUND

Corneal topography refers to the mapping of the surface of the cornea.Corneal topography has been used to diagnose and treat medicalconditions of the eye and is performed in a clinical setting using acorneal topograph, which is a device that projects an illuminationpattern of concentric rings, known as Placido rings, onto the eye. Thetopology of the cornea is then computed based on the reflected pattern.Corneal topographs are large and need to operated by a trainedprofessional, and are therefore unsuitable for use in portable devices,such as head-mounted displays (HMDs).

HMDs are a wearable form of near-eye display (NED) and are sometimesused for displaying content in an augmented reality (AR) or virtualreality (VR) system. Various eye tracking schemes have been applied inHMDs and rely on placing a few light sources outside of the user's fieldof view. In a conventional eye tracking scheme, the cornea is assumed tobe a perfect sphere.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative embodiments are described with reference to the followingfigures.

FIG. 1 shows a near-eye display, in accordance with an embodiment.

FIG. 2 shows a cross-sectional view of near-eye display, in accordancewith an embodiment.

FIG. 3 shows an isometric view of a waveguide assembly, in accordancewith an embodiment.

FIG. 4 shows a simplified representation of a reference pattern whichmay be used to implement one or more embodiments.

FIG. 5 shows a process for generating an illumination pattern by warpinga reference pattern, in accordance with an embodiment.

FIG. 6 shows a process for generating a corneal map, in accordance withan embodiment.

FIG. 7 is a flowchart of a method for generating an illuminationpattern, in accordance with an embodiment.

FIG. 8 is a flowchart of a method for performing corneal topography,according to an embodiment.

FIG. 9 is a flowchart of a method for eye tracking based on a model ofthe surface of a user's cornea, in accordance with an embodiment.

FIG. 10 is a block diagram of a system in which one or more embodimentsmay be implemented.

The figures depict embodiments of the present disclosure for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated may be employed without departing from theprinciples, or benefits touted, of this disclosure.

In the appended figures, similar components and/or features may have thesame reference label. Further, various components of the same type maybe distinguished by following the reference label by a dash and a secondlabel that distinguishes among the similar components. If only the firstreference label is used in the specification, the description isapplicable to any one of the similar components having the same firstreference label irrespective of the second reference label.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, specificdetails are set forth in order to provide a thorough understanding ofcertain inventive embodiments. However, it will be apparent that variousembodiments may be practiced without these specific details. The figuresand description are not intended to be restrictive.

Example embodiments relate to a system and methods for performingcorneal topography in an HMD. In an example embodiment, the topographyof the cornea is determined using reflections (also referred to hereinas glints) produced by illuminating the surface of the eye with a dotpattern. The dot pattern can be generated using a dense array ofilluminators (e.g., 100 or more points of illumination), which can bepositioned within a field of view of the user, i.e., dense, in-fieldillumination. This is in contrast to eye tracking methods that may relyon only a few light sources positioned at the periphery of the user'sfield of view. In-field illumination may offer greater eye-trackingaccuracy than positioning the light sources at the periphery of theuser's field of view. For example, the probability of capturing glintsover all gaze angles of the eye is higher when the light sources arelocated within the user's field of view.

Further, the use of dense illumination in the form of a dot patternenables capturing of many reflections off the eye, providing a rich dataset for use in reconstructing the shape of the cornea for use in eyetracking or other applications. The dot pattern can be projected withoutthe use of a corneal topograph, and a corneal topography procedure canbe automated so as not to require the supervision of a trainedprofessional. In one embodiment, the number of illuminators is reduced(e.g., to a single point source) by generating the illumination patternusing holographic projection or other methods capable of producing anillumination pattern from a point source.

Example embodiments relate to the formation of an illumination patterncomprising an array of dots that are encoded so that the dots areindividually identifiable in a reflected pattern. In one embodiment, theillumination pattern is generated as a non-rectilinear pattern shaped sothat the illumination pattern becomes rectilinear or substantiallyrectilinear when reflected off the surface of the cornea. Making thereflected pattern rectilinear or substantially rectilinear facilitatescomparison of the reflected pattern to a reference pattern, which mayalso be rectilinear.

Example embodiments relate to a system and methods for applying an eyemodel generated using corneal topography. In one embodiment, the eyemodel is used to perform eye tracking by comparing the eye model to datafrom subsequently captured images of the eye. In particular, movementsof the eye can be determined using reflections off the cornea (glintonly) or using glints in combination with tracking of other eye featuressuch as the pupil or iris.

Embodiments of the present disclosure may include or be implemented inconjunction with an artificial reality system. Artificial reality is aform of reality that has been adjusted in some manner beforepresentation to a user, which may include, e.g., a VR, an AR, a mixedreality (MR), a hybrid reality, or some combination and/or derivativesthereof. Artificial reality content may include completely generatedcontent or generated content combined with captured (e.g., real-world)content. The artificial reality content may include video, audio, hapticfeedback, or some combination thereof, and any of which may be presentedin a single channel or in multiple channels (such as stereo video thatproduces a three-dimensional effect to the viewer). Additionally, insome embodiments, artificial reality may also be associated withapplications, products, accessories, services, or some combinationthereof, that are used to, e.g., create content in an artificial realityand/or are otherwise used in (e.g., perform activities in) an artificialreality. The artificial reality system that provides the artificialreality content may be implemented on various platforms, including anNED connected to a host computer system, a standalone NED, a mobiledevice or computing system, or any other hardware platform capable ofproviding artificial reality content to one or more viewers.

FIG. 1 shows an NED 100 according to an embodiment of the presentdisclosure. The NED 100 presents media to a user. Examples of mediapresented by the NED 100 include one or more images, video, and/oraudio. In some embodiments, audio is presented via an external device(e.g., speakers and/or headphones) that receives audio information fromthe NED 100, a console, or both, and presents audio output based on theaudio information. The NED 100 can be configured to operate as a VRdisplay. In some embodiments, the NED 100 is modified to operate as anAR display and/or an MR display.

The NED 100 includes a frame 105 and a display device 110. The frame 105is shaped to enable the NED 100 to be worn in the manner of a pair ofeyeglasses. Thus, the NED 100 is an example of an HMD. The frame 105 iscoupled to one or more optical elements. The display device 110 isconfigured for the user to see content presented by NED 100. In someembodiments, the display device 110 comprises a waveguide displayassembly for directing light from one or more images to an eye of theuser.

The NED 100 further includes optical sensors 120 a, 120 b, 120 c, and120 d. Each of the optical sensors 120 a-120 d can be an image sensorthat includes a pixel cell array comprising an plurality of pixel cells(e.g., a two-dimensional (2D) pixel cell array) and configured togenerate image data representing different fields of view alongdifferent directions toward the user, in particular toward one or morefeatures of a face 135 of the user. For example, the sensors 120 a and120 b may be configured to provide image data representing two fields ofview toward a direction A along the Z axis, whereas sensor 120 c may beconfigured to provide image data representing a field of view toward adirection B along the X axis, and sensor 120 d may be configured toprovide image data representing a field of view toward a direction Calong the X axis. The sensors 120 a and 120 b can be used to capturefront views of the user's facial features, e.g., the nose, eye, lips,the entire face 135, etc. The sensors 120 c and 120 d can be used tocapture side views of the user's facial features. In some embodiments,the optical sensors 120 a-120 d are silicon-based. For example, each ofthe optical sensors 120 a-120 d can be a complementary metal oxidesemiconductor (CMOS) sensor.

The NED 100 can include additional optical sensors (not shown)configured to provide image data representing fields of view away fromthe user, e.g., directed toward the external environment. Suchadditional sensors could be used as input devices to control orinfluence the display content of the NED 100, including to provide aninteractive VR/AR/MR experience to the user. For example, in someembodiments, additional sensors could provide images of the externalenvironment to a location tracking system of the NED 100 to track alocation and/or a path of movement of the user in the externalenvironment. The NED 100 could then update image data provided todisplay device 110 based on, for example, the location and orientationof the user, to provide the interactive experience.

The NED 100 may further include one or more active illuminators 130configured to project light toward the user and/or toward the externalenvironment. Active illuminators are activated using electrical signalsthat cause the illuminators to project light. The light projected can beassociated with different frequency spectrums (e.g., visible light,infrared (IR) light, near infrared (NIR) light, ultra-violet (UV) light,etc.) and can serve various purposes. For example, illuminator 130 mayproject light and/or light patterns in a dark environment (or in anenvironment with little or no IR light, NIR light, etc.) to assistsensors 120 a-120 d in capturing 2D or 3D images of the face 135 of theuser. The illumination patterns formed by the illuminator 130 can beplaced within a field of view of the user and can be reflected off oneor more eyes of the user. In some embodiments, a separate illuminator130 is used for illuminating the left and right eyes. The illuminationpatterns produce light that can be reflected off an external surface ofthe eye (e.g., the cornea) or an internal surface (e.g., the retina).Images of the reflected patterns captured by the sensors 120 a-120 d canbe used to perform biometric authentication as well as eye tracking(e.g., determining an eye gaze of the user).

Another active illuminator (not shown) could be used to project lightand/or light patterns for capturing images of objects in the externalenvironment. The images of the objects could be used, for example, totrack the head movement or location of the user based on distances tothe objects, where the distances are determined by the NED 100 using thecaptured images. Optical sensors, including the sensors 120 a-120 d, canbe operated in a first mode for 2D sensing and in a second mode for 3Dsensing. Captured 2D images can be merged with other captured 2D imagesor merged with captured 3D images to provide more robust tracking offacial features as well as tracking of the location of the user, thehead movement of the user, etc.

FIG. 2 shows a cross section of an NED 200. The NED 200 may correspondto the NED 100. The NED 200 includes a frame 205 and a display device210 with at least one waveguide assembly 250. The display device 210 isoperable to present image content to the user and includes the waveguideassembly 250, which is configured to direct light from one or moreimages to an eye 220 of the user. When placed into an operative positionwith respect to the user, e.g., when the user wears the NED 200, the NED200 forms an exit pupil 230 at a location where the eye 220 ispositioned in an eyebox region. For purposes of illustration, FIG. 2shows the cross section associated with a single eye 220 and a singlewaveguide assembly 250, but a second waveguide assembly can be used fora second eye of the user.

The waveguide assembly 250 is configured to direct the image light tothe eye 220 through the exit pupil 230. The waveguide assembly 250 maybe composed of one or more materials (e.g., plastic, glass, etc.) withone or more refractive indices. In some embodiments, the NED 200includes one or more optical elements between the waveguide assembly 250and the eye 220. The waveguide assembly 250 may be composed of one ormore materials with one or more refractive indices that effectivelyminimize the weight and widen a field of view (FOV) of the NED 200.

The NED 200 can include one or more optical elements (not shown) betweenthe waveguide assembly 250 and the eye 220. The optical elements mayform an optics system that acts to, e.g., correct aberrations in imagelight emitted from the waveguide assembly 250, magnify image lightemitted from the waveguide assembly 250, some other optical adjustmentof image light emitted from the waveguide assembly 250, or somecombination thereof. Example optical elements include an aperture, aFresnel lens, a convex lens, a concave lens, a filter, a reflector, orany other suitable optical element that affects image light.

FIG. 3 shows an isometric view of a waveguide assembly 300. In someembodiments, the waveguide assembly 300 is a component of a displaydevice in an NED (e.g., waveguide assembly 250). In other embodiments,the waveguide assembly 300 is a separate component located along anoptical path between the display device and an eye 390 of the user. Thewaveguide assembly 300 includes an optics system 310, an outputwaveguide 320, a controller 330, a coupling element 350, a directingelement 360, and a decoupling element 365.

The waveguide assembly 300 receives emitted light 355 from a displaydevice (not shown) and processes the emitted light 355 through theoptics system 310 to generate image light 357. The emitted light 355 isgenerated by light emitters of the display device. A light emitter canbe a light emitting diode (LED), a micro light emitting diode (mLED), avertical-cavity surface-emitting laser (VCSEL), a photonics integratedcircuit (PIC), etc. The light emitters can be organized in aone-dimensional (1D) or 2D array. The emitters can be grouped to formpixels of the display device. For example, an individual pixel mayinclude at least one red emitter, at least one green emitter, and atleast one blue emitter.

The optics system 310 comprises one or more optical elements (e.g., oneor more lenses, a scanning mirror, etc.) that perform a set of opticalprocesses, including, but not restricted to, focusing, combining,collimating, transforming, conditioning, and scanning processes on theemitted light from the display.

The output waveguide 320 is an optical waveguide that directs the imagelight 357 to form output image light 340 to the eye 390 of the user. Theoutput waveguide 320 may be composed of one or more materials thatfacilitate total internal reflection of the image light 357. Forexample, the output waveguide 320 may be composed of silicon, plastic,glass, and/or polymers. The output waveguide 320 receives the imagelight 357 at one or more coupling elements 350, and guides the imagelight 357 to the directing element 360. The coupling element 350 mayinclude, e.g., a diffraction grating, a holographic grating, some otherelement that couples the image light 357 into the output waveguide 320,or some combination thereof. For example, in embodiments where thecoupling element 350 includes a diffraction grating, the pitch of thediffraction grating can be chosen such that total internal reflectionoccurs and the image light 357 propagates internally toward thedecoupling element 365.

The directing element 360 redirects the image light 357 to decouplingelement 365 such that the image light 357 is decoupled out of outputwaveguide 320 via decoupling element 365. Directing element 360 is partof, or affixed to, a first side 370-1 of output waveguide 320.Decoupling element 365 is part of, or affixed to, a second side 370-2 ofoutput waveguide 320, such that directing element 360 is opposed to thedecoupling element 365. The second side 370-2 represents a plane alongan x-dimension and a y-dimension. Directing element 360 and/ordecoupling element 365 may be, e.g., a diffraction grating, aholographic grating, one or more cascaded reflectors, one or moreprismatic surface elements, and/or an array of holographic reflectors.

The controller 330 drives the light emitters of the display device togenerate the emitted light 355. The controller 330 can also analyzeimage data captured by the optical sensors 120 a-120 d. The controller330 may instruct the display device to form the emitted light 355 basedon the analysis of the image data captured by the optical sensors 120a-130 d. For example, the emitted light 355 may form interactive imagecontent that is updated in response to the user's eye movements. Thecontroller 330 can also control the optics system 310 to perform the oneor more optical processes. For example, the controller 330 mayreposition one or more lenses of the optics system 310 to adjust a focusof the image light 357.

In addition to controlling the display device and the optics system 310,the controller 330 can perform biometric authentication of the userbased on illumination light 380 generated by the illuminator 130 andcorresponding reflected light 382 captured by the optical sensors 120a-120 d. In some embodiments, the illuminator 130 and/or the opticalsensors 120 a-120 d may be part of, or affixed to the output waveguide320, e.g., on the second side 370-2. Alternatively, the illuminator 130and/or the optical sensors 120 a-120 d can be located in variouspositions throughout a head-mounted assembly that incorporates thewaveguide assembly 300, e.g., in the positions shown in FIG. 1. Further,although described as being generated by a separate light source (i.e.,illuminator 130), in some embodiments illumination light may begenerated by one or more light emitting elements of the display device.For example, an illumination pattern can be generated using VCSELs ofthe display device.

The controller 330 may instruct the illuminator 130 to generate one ormore images that contain a light pattern that is reflected off the eye390 as reflected light 382 toward the sensors 120 a-120 d. The reflectedlight 382 can be directly reflected from the eye 390 into the sensors120 a-120 d. Alternatively, light reflected off the eye can be furtherreflected off an optical surface before reaching a sensor. For example,the light may be reflected off an optical surface of the waveguideassembly 300.

As mentioned earlier in the discussion of FIG. 1, optical sensors can beoperated to perform 2D or 3D sensing. For 2D sensing, the opticalsensors 120 a-120 d can be operated to generate pixel data representingan intensity of the reflected light 382. For 3D sensing, the opticalsensors 320 a-320 d can be operated to generate pixel data representinga time-of-flight measurement (e.g., a difference between a time at whichthe illumination light 380 is output by the illuminator 120 and a timeat which the reflected light 382 is received by the optical sensors 120a-120 d). The optical sensors 120 a-120 d can be operated to perform 2Dand 3D sensing at different times, and to provide the 2D and 3D imagedata to the controller 330 or to a remote console (not shown) that ispart of, or communicatively coupled to, a head-mounted assembly. Thecontroller 330 or remote console can form a 2D or 3D model of one ormore facial features using information derived from the 2D and 3D imagedata. The model may include, for example, a model of the surface of thecornea of the eye 390.

FIG. 4 shows a simplified representation of a reference pattern 400which may be used to implement one or more embodiments. The referencepattern 400 comprises a dense array of geometric features. In theembodiment of FIG. 4, the geometric features are formed by dots that areencoded such that the dots can be individually identified in a reflectedpattern. A geometric feature may correspond to a single dot or a groupof dots. Each dot represents a point of illumination and may be uniquelyencoded. The number of points of illumination may vary and is preferablyat least one hundred. In one embodiment, the number of points ofillumination is at least one thousand. For simplicity, the dots areshown in black and white, where black indicates illumination points thatare turned off and white indicates illumination points that are turnedon. However, grey-scale and color patterns are also possible.

Various encoding schemes may be used for encoding the reference pattern400. For example, the reference pattern 400 may be binary encoded, wherea value of 1 corresponds to a state of “on” and a value of 0 correspondsto “off,” i.e., a binary data indicating whether a geometric feature ispresent or absent at a particular position in the reference pattern.Alternatively, the reference pattern may be encoded based on brightnessvalue, e.g., based on a gray-scale value indicating a brightness of aparticular geometric feature. In some embodiments, the reference patternmay be a time varying pattern that is temporally encoded or wavelengthencoded. For example, a first pattern of dots that output light of afirst wavelength and a second pattern of dots that output light ofsecond wavelength may be projected onto the user's eye in alternatingframes or in a sequence of frames, in which case wavelength data mayindicate which wavelength a particular geometric feature has. Similarly,temporal data may be used to indicate changes in the geometric featuresover time. Thus, the reference pattern does not have to be a staticpattern.

As shown in FIG. 4, the reference pattern 400 is rectilinear, with thedots being arranged along a 2D grid of parallel lines. To facilitateidentification, the dots may be regularly spaced apart andnon-overlapping. The reference pattern 400 may be generated randomly orpseudo-randomly to meet certain criteria. For example, the dots may beconstrained by the rule that in any 3×3 neighborhood or block of dots,there is at least one bright dot and at least one dark dot, such that no3×3 block is completely bright or completely dark. To facilitateidentification of the dots in a reflected pattern, the reference patternmay be generated to have a desired autocorrelation. In one embodiment,the desired autocorrelation is a “spikey” autocorrelation function, inwhich most of the values produced by convolving the reference patternwith a delayed version of the reference pattern (e.g., the reflectedpattern) are relatively small such that the peak values of theconvolution (corresponding to locations where the delayed patternmatches the original reference pattern) can be used to easily determinehow much the delayed version has been shifted relative to the originalreference pattern.

In other words, when the reflected pattern is received, it is notnecessarily aligned with the reference pattern. There may be an unknownshift in both the x and y directions between the two patterns. If thereference pattern has a desired (e.g., “spikey”) autocorrelationfunction, it can be easily determined whether alignment has beenreached—i.e., when the dot product of the reflected pattern and thereference pattern results in a peak value. In fact, the autocorrelationfunction of the reference pattern can be chosen to have predictablevalues at non-zero x and y offsets (e.g. increasing values toward thepeak), such that even when the x and y shifts are non-zero, dot productsof the reflected pattern and the reference pattern can reveal the valueof the relative x and y shifts between the two patterns. Thus, choosinga desired autocorrelation function can greatly facilitate the process ofmapping the reflected pattern to the reference pattern.

In some embodiments, the reference pattern 400 may include non-randomelements such as fiducial markers. For example, at least some of thedots may correspond to a 2D barcode or a reference image. The use offiducials may facilitate identification of dots in the reflectedpattern. Additionally, dots do not have to be circular or even round,but may instead have other shapes such as square or rectangular.

Because the surface of the eye is not flat, projecting a rectilinearpattern such as the reference pattern 400 onto the eye would result in areflected pattern that is distorted, making identification of dotsdifficult. To overcome this problem, the illumination pattern may beformed by “pre-warping” (i.e., intentionally distorting) the referencepattern so that the reflected pattern becomes a rectilinear pattern ofapproximately the same shape as the original, rectilinear referencepattern, e.g., rectilinear (if the shape of the user's eye matchesexactly the shape of the eye model used to generate the illuminationpattern) or substantially rectilinear (if the shape of the user's eyedeviates from the eye model used to generate the illumination pattern).Thus, the reflected pattern may also include a set of dots arrangedalong a 2D grid, with a majority of the dots being located alongstraight lines in the x and y directions. FIG. 5 illustrates thispre-warping concept. In FIG. 5, a reference pattern 500 has been warpedto form an illumination pattern 500′. For simplicity, the pattern ofbright and dark dots is omitted in FIG. 5. The pre-warping of thereference pattern may be performed based on a model of a typical orideal human eye, e.g., assuming an average corneal radius located at anaverage position of a user's eye as seen by an optical sensor when theHMD is worn, with the assumption that the eye is completely spherical.

FIG. 6 shows a process for generating a corneal map 600 according to anembodiment. The process begins with capturing an eye image 610 thatincludes at reflected pattern corresponding to at least a portion of anillumination pattern. The illumination pattern is projected onto asurface of the user's eye, including onto a cornea 615. The eye image610 may be captured using an optical sensor, e.g., the sensors 120 a-120d discussed in connection with FIGS. 1 and 3. The eye image 610 mayinclude other features of the user's face that are not needed forcorneal topography. These extraneous features may be removed by applyingimage processing to the eye image 610, e.g., cropping the eye image 610to remove portions of the eye image that are outside of the area of thecornea.

The corneal portion of the eye image 610 includes a reflected pattern620, which corresponds to at least a portion of an illumination patternthat is reflected off the exterior surface of the cornea 615. Thereflected pattern 620 includes a pattern of dots, with a darker areacorresponding to an iris region 625. In some embodiments, the projectedillumination pattern is larger than the area of the cornea so that thereflected pattern includes only part of the illumination pattern. Inother embodiments, the projected illumination pattern may be smallerthan the area of the cornea so that the entire illumination pattern isrepresented in the reflected pattern. The reflected pattern 620 can becompared to a reference pattern to determine the shape of the cornea andform the corneal map 600. Differences between the reference pattern andthe reflected pattern may indicate deviations of the user's cornea froma modeled or ideal corneal shape.

Example methods will now be described. The methods can be performedusing a system according to an embodiment, e.g., using an HMD with anilluminator capable of projecting an illumination pattern onto a surfaceof the user's eye, and one or more processors or control units operableto perform the various processing steps.

FIG. 7 is a flowchart of a method 700 for generating an illuminationpattern, according to an embodiment. The method 700 transforms areference pattern to produce a warped version of a reference pattern foruse as an illumination pattern. At step 710, a reference pattern isgenerated to include encoded features such as a rectilinear pattern ofdots.

At step 720, the reference pattern is projected onto an eye model, whichmay correspond to a typical or ideal human eye.

At step 730, a pattern reflected off a cornea of the eye model isdetermined using computer simulation, which may include, for example,ray tracing to locate the corresponding positions of dots in thereflected pattern.

At step 740, the pattern determined in step 730 is stored as anillumination pattern in a computer-readable memory that can subsequentlybe accessed for projecting the illumination pattern onto a user's eye.The illumination pattern includes geometric features corresponding tothose of the reference pattern, and thus the same number of points ofillumination, but shaped differently. The illumination pattern may beencoded for storage using the same encoding scheme as the referencepattern.

FIG. 8 is a flowchart of a method 800 for performing corneal topography,according to an embodiment. At step 810, an illumination patternincluding encoded features is projected onto a user's eye. Theillumination pattern is based on a reference pattern (e.g., a warpedversion of a rectilinear reference pattern) and corresponds to selectiveillumination of dots arranged along a two-dimensional grid.

At step 820, an image of the illumination pattern as reflected off thesurface of the user's cornea is captured using one or more opticalsensors.

At step 830, a reflected pattern is identified by based on glints in thereflected image. The identification can be based on characteristics ofthe glints such as size, position relative to neighboring glints (e.g.,distance or orientation), intensity, etc. Various identificationtechniques may be applied depending on how the illumination pattern wasencoded.

At step 840, the reflected pattern is mapped to a reference pattern toobtain an aligned reflected pattern. The mapping may include determiningan orientation of the reflected pattern relative to the referencepattern, determining correspondences between dots in the reflectedpattern and dots in the reference pattern, and determine how much thereflected pattern has been shifted (e.g., in an x direction and/or a ydirection) relative to the reference pattern.

Steps 820 to 840 may be repeated to capture, identify, and alignreflected patterns in multiple images corresponding to the user's eye indifferent positions. For example, the user may be instructed to look inone or more directions (e.g., left and right) to capture reflectionsover the entire corneal surface. Information from one or more capturedimages can then be applied to produce a model of the user's eye in step850.

At step 850, the model of the user's eye, which includes a topography ofthe cornea, is calculated using the information from the capturedimage(s), e.g., by comparing an aligned reflected pattern to a referencepattern that corresponds to the illumination pattern used in step 810.The calculation may include solving for the 3D shape of the cornea todetermine the topography of the cornea.

FIG. 9 is a flowchart of a method 900 for eye tracking based on a modelof the surface of the user's cornea, according to an embodiment. Forpurposes of eye tracking, the cornea does not need to be continuouslyremapped (e.g., on a frame-by-frame basis). Instead, once a model of thesurface of the cornea has been calculated (e.g., during a calibrationphase in which the method 800 is performed) the model can be fitted tonew observations about the position of the cornea. The method 900 can beperformed using the same illumination pattern as that which waspreviously used to generate a model of the user's eye. Alternatively, aless resource intensive illumination pattern may be used. Thus, at step910, a lower resolution illumination pattern is projected onto theuser's eye. The lower resolution illumination pattern may, for example,correspond to the earlier illumination pattern with one or moreillumination points deactivated in order to conserve power. Thus, thelower resolution illumination pattern may comprise a subset of thegeometric features of the earlier illumination pattern. In someembodiments, the lower resolution illumination pattern may include thesame illumination points, but with lower intensities that are stillsufficient for producing glints.

At step 920, images of the lower resolution illumination pattern arecaptured by one or more optical sensors as the eye moves across theuser's field of view.

At step 930, reflected patterns are identified based on glints in thereflected images.

At step 940, a previously calculated model of the corneal surface of theuser's eye is applied to the reflected patterns to track the movement ofthe eye in 3D space (e.g., with six degrees of freedom) by estimatingthe shape and pose of the eye using techniques such as non-linear leastsquares analysis. Thus, the previously calculated model of the cornealsurface is used as a reference against which the reflected patterns areanalyzed to determine a correspondence between movement of the eye andchanges in the reflected patterns. For example, the analysis maydetermine how much the eye has shifted relative to the position of theeye when the model of the corneal surface was generated. In someembodiments, glint information may be combined with information obtainedfrom the images about the position of one or more eye features such asthe pupil or iris. For example, a shift in the corneal surface appliedto a plane of the pupil to calculate a location in 3D space for a centerof the pupil. Thus, eye tracking may be performed using glints only orglints in combination with an eye feature.

FIG. 10 is a block diagram of a system 1000 in which one or moreembodiments may be implemented. The system 1000 includes an HMD 1010, acontrol unit 1030, and an input/output (I/O) interface 1040. The HMD1010 includes a display device 1012, a waveguide assembly 1014, at leastone proximity sensor 1016, at least one illuminator 1018, at least oneoptical sensor 1020, at least one position sensor 1022, and an inertialmeasurement unit 1024.

The display device 1012 includes a display screen for presenting visualmedia, such as images and/or video, to a user. The display device 1012may correspond to one of the earlier described display devices, e.g.,the display device 110 or the display device 210. In addition to visualmedia, the HMD 1010 may include an audio output device (not shown) forpresenting audio media to the user, e.g., in conjunction with thepresentation of the visual media.

The waveguide assembly 1014 is part of, or affixed to, the displaydevice 1012 and may correspond to the waveguide assembly 250 in FIG. 2or the waveguide assembly 300 in FIG. 3. The waveguide assembly 1014 mayalso include an optical substrate for directing a reflected illuminationpattern toward the optical sensor(s) 1020.

The proximity sensor 1016 can be any sensor capable of detecting thatthe user is wearing the HMD 1010. For example, the proximity sensor 1016can be a simple mechanical switch that is activated when the user's headis pressed against a frame of the HMD 1010. The proximity sensor 1016can be a resistive or capacitive touch sensor configured to detectcontact with the user's head based on electrical measurements. In someembodiments, the proximity sensor 1016 is an optical sensor. Forpurposes of detecting whether the user is wearing the HMD 1010, theproximity sensor 1016 does not have to be an imaging sensor. Forexample, the proximity sensor 1016 can be a passive IR sensor thatdetects the user's presence based on a change in the intensity ofinfrared light emitted from nearby objects. Alternatively, the proximitysensor 1016 can be an active IR sensor that emits infrared light anddetects the resulting infrared reflections. The proximity sensor 1016can be used to verify the user's presence before initiating cornealtopography, eye tracking, and/or biometric authentication.

The illuminator 1018 is an electrically triggered light source thatgenerates light for use in connection with corneal topography, eyetracking, and/or biometric authentication. The illuminator 1018 maycorrespond to the illuminator 130. The illuminator 1018 can be placed ina frame of the HMD 1010 or integrated into an optical component such asthe display device 1012 or the waveguide assembly 1014.

The optical sensor 1020 is an image sensor configured to capture 2Dand/or 3D image data, for example, a 2D image of the user's eye. Theoptical sensor 1020 may correspond to one of the optical sensors 120a-120 d.

The inertial measurement unit 1024 is an electronic device thatgenerates data indicating an estimated position of the HMD 1010 relativeto an initial position of HMD 1010, based on measurement signalsreceived from the position sensor 1022. The measurement signals caninclude, for example, signals indicative of roll, pitch, yaw, oracceleration.

The I/O interface 1040 is a device that allows the user to send actionrequests to the control unit 1030. An action request is a request toperform a particular action. For example, an action request may be tostart or end an application or to perform a particular action within theapplication.

The control unit 1030 is configured to direct the operation of the HMD1010 and can implement any of the controller functions described earlierincluding, for example, selecting content for presentation on thedisplay device 1012 and activating the illuminator 1018 to project anillumination pattern. The control unit 1030 includes an authenticationmodule 1032, a tracking module 1034, one or more processors 1036, an eyeinformation data store 1037, and an application store 1039. The controlunit 1030 can include components that are integrated into the HMD 1010.In some embodiments, one or more components of the control unit 1030 areremotely located. For example, the eye information data store 1037 canbe located on a remote server or distributed between a memory of thecontrol unit 1030 and a remote server.

The eye information data store 1037 stores eye information for a user ofthe HMD 1010. The eye information may include a 3D eye model thatincludes a topography of a cornea of the user, e.g., a model of thecorneal surface generated using one or more of the techniques describedearlier. The eye model stored in the eye information data store 1037 canbe compared to an eye model generated using the HMD 1010 to authenticatethe user or perform eye tracking. The eye information data store 1037may also store one or more reference patterns and one or morecorresponding illumination patterns.

The application store 1039 stores one or more applications for executionby the control unit 1030. An application is a set of instructionsexecutable by a processor, for example instructions that cause theprocessor to generate content for presentation to the user on thedisplay device 1012. Examples of applications include: gamingapplications, conferencing applications, video playback application, orother suitable applications.

The authentication module 1032 can be implemented in hardware and/orsoftware and is configured to perform a biometric authentication of theuser through activation of the illuminator 1018 and analysis of theresulting image data captured by the optical sensor 1020. Execution ofan application or access to certain functions of an application in theapplication store 1039 can be conditioned upon successful authenticationof the user.

The tracking module 1034 can be implemented in hardware and/or softwareand is configured to track changes in the position of the HMD 1010and/or the position of the user's facial features. For example, thetracking module 1034 may track the movements of the HMD 1010 andcorrelate the HMD movements to movement of the user's head. The trackingmodule 1034 may also track the user's eye movements using a stored eyemodel.

The processor 1036 executes instructions from applications stored in theapplication store 1039 and/or instructions provided to the processor1036 by the authentication module 1032 or the tracking module 1034. Theprocessor 1036 can receive various items of information used in theapplications. This includes, for example, position information,acceleration information, velocity information, captured images, and/orreflected patterns. Information received by processor 1036 may beprocessed to produce instructions that determine content presented tothe user on the display device 1012.

The foregoing description of the embodiments of the disclosure has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the disclosure to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of thedisclosure in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, and/or hardware.

Steps, operations, or processes described may be performed orimplemented with one or more hardware or software modules, alone or incombination with other devices. Although the steps, operations, orprocesses are described in sequence, it will be understood that in someembodiments the sequence order may differ from that which has beendescribed, for example with certain steps, operations, or processesbeing omitted or performed in parallel or concurrently. In someembodiments, a software module is implemented with a computer programproduct comprising a computer-readable medium containing computerprogram code, which can be executed by a computer processor forperforming any or all of the steps, operations, or processes described.

Embodiments of the disclosure may also relate to an apparatus forperforming the operations described. The apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

The language used in the specification has been principally selected forreadability and instructional purposes, and it may not have beenselected to delineate or circumscribe the inventive subject matter. Itis therefore intended that the scope of the disclosure be limited not bythis detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thedisclosure, which is set forth in the following claims.

What is claimed is:
 1. A method, comprising: projecting an illuminationpattern onto an eye of a user wearing a head-mounted assembly, whereinprior to being projected, the illumination pattern was derived from areference pattern of dots arranged along a two-dimensional grid, andwherein: the dots form geometric features, the reference pattern is arectilinear pattern in which the two-dimensional grid has straight linesthat define rows and columns in which the dots are arranged, theillumination pattern is a non-rectilinear pattern that, when reflectedoff the eye of the user, produces a reflected pattern that is asubstantially rectilinear approximation of the reference pattern, andthe non-rectilinear pattern corresponds to the dots of the referencepattern rearranged according to a transformation of the two-dimensionalgrid to include non-straight lines; capturing, using an image sensor, areflected image produced by reflection of the illumination pattern offthe eye; identifying the reflected pattern based on glints in thereflected image, wherein the reflected pattern is directionally shiftedrelative to the reference pattern; generating an aligned reflectedpattern based on mapping the reflected pattern to the reference pattern,wherein mapping the reflected pattern to the reference pattern comprisesdetermining how much the reflected pattern has been shifted; andcalculating an eye model including a topography of a cornea of the eye,the calculating comprising comparing the aligned reflected pattern tothe reference pattern to determine a deviation between a shape of thecornea and a shape of a model cornea based on a difference between thealigned reflected pattern and the reference pattern.
 2. The method ofclaim 1, wherein the illumination pattern is obtained through reflectionof the reference pattern off a computer simulated eye such that thereference becomes non-rectilinear due to a shape of the computersimulated eye.
 3. The method of claim 1, wherein the reference patternis encoded using one or more of: binary data indicating whether ageometric feature is present or absent at a particular position withinthe reference pattern; brightness data indicating a brightness of aparticular geometric feature of the reference pattern; wavelength dataindicating a wavelength of a particular geometric feature of thereference pattern; and temporal data indicating changes in geometricfeatures of the reference pattern over time.
 4. The method of claim 1,wherein the illumination pattern includes at least 100 points ofillumination.
 5. The method of claim 1, further comprising: calculatingthe eye model using multiple reflected patterns produced by reflectionof the illumination pattern off the eye in different positions.
 6. Themethod of claim 1, wherein the illumination pattern is projected usingan illuminator located on a display device of the head-mounted assembly,the illuminator being positioned with a field of view of the user. 7.The method of claim 1, further comprising: tracking a movement of theeye by referencing the eye model against a second reflected imageproduced by reflection of a lower resolution illumination pattern,wherein the lower resolution illumination pattern comprises a subset ofdots in the illumination pattern or the same dots in the illuminationpattern but with lower intensity.
 8. The method of claim 7, wherein thelower resolution illumination pattern comprises the same dots in theillumination pattern but with lower intensity.
 9. A head-mountedassembly, comprising: an illuminator configured to project anillumination pattern onto an eye of a user wearing the head-mountedassembly, wherein prior to being projected by the illuminator, theillumination pattern was derived from a reference pattern of dotsarranged along a two-dimensional grid, and wherein: the dots formgeometric features, the reference pattern is a rectilinear pattern inwhich the two-dimensional grid has straight lines that define rows andcolumns in which the dots are arranged, the illumination pattern is anon-rectilinear pattern that, when reflected off the eye, produces areflected pattern that is a substantially rectilinear approximation ofthe reference pattern, and the non-rectilinear pattern corresponds tothe dots of the reference pattern rearranged according to atransformation of the two-dimensional grid to include non-straightlines; an image sensor configured to capture a reflected image producedby reflection of the illumination pattern off the eye; and one or moreprocessors configured to: identify the reflected pattern based on glintsin the reflected image, wherein the reflected pattern is directionallyshifted relative to the reference pattern, generate an aligned reflectedpattern based on mapping the reflected pattern to the reference pattern,wherein mapping the reflected pattern to the reference pattern comprisesdetermining how much the reflected pattern has been shifted, andcalculate an eye model including a topography of a cornea of the eye,wherein to calculate the eye model, the one or more processors comparethe aligned reflected pattern to the reference pattern to determine adeviation between a shape of the cornea and a shape of a model corneabased on a difference between the aligned reflected pattern and thereference pattern.
 10. The head-mounted assembly of claim 9, wherein theillumination pattern is obtained through reflection of the referencepattern off a computer simulated eye such that the reference patternbecomes non-rectilinear due to a shape of the computer simulated eye.11. The head-mounted assembly of claim 9, wherein the reference patternis encoded using one or more of: binary data indicating whether ageometric feature is present or absent at a particular position withinthe reference pattern; brightness data indicating a brightness of aparticular geometric feature of the reference pattern; wavelength dataindicating a wavelength of a particular geometric feature of thereference pattern; and temporal data indicating changes in geometricfeatures of the reference pattern over time.
 12. The head-mountedassembly of claim 9, wherein the illumination pattern includes at least100 points of illumination.
 13. The head-mounted assembly of claim 9,wherein the one or more processors are further configured to calculatethe eye model using multiple reflected patterns produced by reflectionof the illumination pattern off the eye in different positions.
 14. Thehead-mounted assembly of claim 9, wherein the illuminator is located ona display device of the head-mounted assembly, the illuminator beingpositioned with a field of view of the user.
 15. The head-mountedassembly of claim 9, wherein the one or more processors are furtherconfigured to track a movement of the eye by referencing the eye modelagainst a second reflected image produced by reflection of a lowerresolution illumination pattern, wherein the lower resolutionillumination pattern comprises a subset of dots in the illuminationpattern or the same dots in the illumination pattern but with lowerintensity.
 16. The head-mounted assembly of claim 15, wherein the lowerresolution illumination pattern comprises the same dots in theillumination pattern but with lower intensity.
 17. A method, comprising:encoding a set of geometric features into a reference pattern, thegeometric features including dots arranged along a two-dimensional grid,wherein the reference pattern is a rectilinear pattern in which thetwo-dimensional grid has straight lines that define rows and columns inwhich the dots are arranged; transforming, by a processor of a computer,the reference pattern into an illumination pattern, wherein theillumination pattern is a non-rectilinear pattern that, when reflectedoff an eye of a person, produces a substantially rectilinearapproximation of the reference pattern, the transforming comprising:projecting the reference pattern onto a computer simulated eye, anddetermining a reflected pattern produced through reflection of thereference pattern off a cornea of the computer simulated eye, whereinthe reflected pattern corresponds to the dots of the reference patternrearranged according to a transformation of the two-dimensional grid toinclude non-straight lines; and storing the reflected pattern as theillumination pattern in a computer-readable memory, thecomputer-readable memory being accessible to read the illuminationpattern in connection with projecting the illumination pattern onto theeye of the person.
 18. The method of claim 17, further comprising:encoding the reference pattern using one or more of: binary dataindicating whether a geometric feature is present or absent at aparticular position within the reference pattern; brightness dataindicating a brightness of a particular geometric feature of thereference pattern; wavelength data indicating a wavelength of aparticular geometric feature of the reference pattern; and temporal dataindicating changes in geometric features of the reference pattern overtime.
 19. The method of claim 17, wherein the illumination patternincludes at least 100 points of illumination.